Skills hopefully acquired at the end of lecture

Take a single variable in a dataset.

Visualize to learn more about it. Key cases of interest: * Categorical variables (in lecture 2 we called these N/O) * Continuous variables (in lecture 2 we called these Q) * Exploring typical values * Exploring and dealing with unusual values

(Up next class: covariation with two variables)

Roadmap of lecture

• Introduce three key ideas in data visualization
  • exploration vs. production
  • headline vs. sub-messages
  • iteration
• introduce datasets for this lecture
• Categorical variables (in lecture 2 we called these N/O)
• Continuous variables (in lecture 2 we called these Q)
  • Exploring typical values
  • Exploring and dealing with unusual values

What is exploratory data analysis?

Data visualization has two distinct goals

1. exploration for you to learn as much as possible
2. production for you to teach someone else what you think the key lessons are

Exploration vs. production

• When you are in exploration mode, you will look at lots of patterns and your brain filters out the noise
• Production mode is like putting a cone on your dog.
• You are deliberately limiting the reader’s field of vision such that they see the key messages from the plot and avoid too many distractions

“A Sunday on La Grande Jatte” by Seurat

“A Sunday on La Grande Jatte” by Seurat

Headline vs submessages

• Headline is what you see first when you look at the painting or you look at the plot.
• Submessages are what you see later, on closer inspection
• Often, the most interesting patterns in the data are ones that you don’t see right away when you make the very first plot.

Iterating on plot design

“Make dozens of plots”

Quoctrung Bui, former 30538 guest lecturer and former Harris data viz instructor

Iterating on plot design

What does he mean?

• The first plot you make will never be the one you should show
• As you are generating graphs for yourself in exploration mode, you will produce many candidates that could end up being used in production mode
• As a rule of thumb, you should try out at least three different plotting concepts (marks)
• Within each concept, you will need to try out several different encodings

Summary:

1. Decide if you are trying to explore the data or produce a plot for someone else
2. For any given plot, look closely (like Seurat) beyond just the headline
3. Iterate

Introduction to data

• Most of our visualization lectures are based on the University of Washington textbook, but the textbook doesn’t have enough material on exploratory data analysis. So we are supplementing with
  • Data Visualization
  • Exploratory Data Analysis material in the R for Data Science textbook (with the code translated to Altair)
• diamonds, mpg are from “Exploratory Data Analysis”
• movies (also used last lecture) is from the UW textbook
• penguins is from “Data Visualization”

Categorical variables: roadmap

• introduce diamonds
• show table
• show bar graph

introduce dataset diamonds

import pandas as pd
from plotnine.data import diamonds
diamonds.shape

(53940, 10)

diamonds.head()

diamonds data dictionary

Summarizing cut in a table

diamonds_cut = diamonds.groupby('cut').size()
diamonds_cut

cut
Fair          1610
Good          4906
Very Good    12082
Premium      13791
Ideal        21551
dtype: int64

Summarizing with a bar graph (code)

#moves the index (`cut`) into a normal column so Altair can read it.
diamonds_cut = diamonds_cut.reset_index().rename(columns={0:'N diamonds'})

# extract ordering of `cut` as a Python list to use for sorting
cut_order = diamonds['cut'].cat.categories.tolist()

alt.Chart(diamonds_cut).mark_bar().encode(
    alt.X('cut:O', title = "Cut", sort=cut_order),
    alt.Y('N diamonds:Q', title = "Count")
).properties(width=640, height=360).configure_axis(
    labelFontSize=18,
    titleFontSize=20
)

Note: we have included syntax to modify graph properties here. Going forward our .qmd source code uses these throughout, but we will omit in the slides for the sake of space.

Summarizing with a bar graph (plot)

Categorical variables: summary

• This section is very brief because there’s basically only one good way to plot categorical variables with a small number of categories and this is it.
  • You can use mark_point() instead of mark_bar(), but overall, there’s a clear right answer about how to do this.
• We include this material mainly to foreshadow the fact that we will do a lot on categorical variables in the next lecture when we get to “Exploring Co-variation”

Continuous variables: roadmap

• Binning + histograms using movies
• Histograms and density plots using penguins
• Exploring carat size using diamonds

Remark: The skills are absolutely fundamental and so we will intentionally be a bit repetitive.

Recall: movies dataset

movies_url = 'https://cdn.jsdelivr.net/npm/vega-datasets@1/data/movies.json'
movies = pd.read_json(movies_url)
movies.head()

Histogram using mark_bar()

• Rotten Tomatoes ratings are determined by taking “thumbs up” and “thumbs down” judgments from film critics and calculating the percentage of positive reviews.
• This is a continuous measure, but we can bin it to create a histogram of frequencies

Histogram using mark_bar()

hist_rt = alt.Chart(movies_url).mark_bar().encode(
    alt.X('Rotten_Tomatoes_Rating:Q', bin=alt.BinParams(maxbins=20), title = "Rotten Tomatoes Rating (%)"),
    alt.Y('count():Q', title = "Count")
)
hist_rt

Discussion question: what are the headline and sub-messages?

Histogram of IMDB ratings

IMDB ratings are formed by averaging scores (ranging from 1 to 10) provided by the site’s users.

hist_imdb = alt.Chart(movies_url).mark_bar().encode(
    alt.X('IMDB_Rating:Q', bin=alt.BinParams(maxbins=20), title = "IMDB Ratings"),
    alt.Y('count():Q', title = "Count")
)
hist_imdb

Side-by-side

hist_rt | hist_imdb

Discussion question: compare the two ratings distributions. If your goal for the headline of the graph is about differentiating between good and bad movies, which rating is more informative?

Introducing the penguins dataset

url = ("https://raw.githubusercontent.com/mcnakhaee/palmerpenguins/master/palmerpenguins/data/penguins.csv")
penguins = pd.read_csv(url)
penguins.head()

Histogram with steps of 200

• We previously picked the maximum number of equally-spaced bins (BinParams(maxbins=20)) and let altair choose “nice”-looking bin widths for the histogram
• Alternatively, we can manually control the bin width using step

Histogram with steps of 200

alt.Chart(penguins).mark_bar().encode(
    alt.X('body_mass_g:Q', bin=alt.BinParams(step=200), title = "Body Mass (g)"),
    alt.Y('count():Q', title = "Count")
)

Histogram step parameter

step=20 vs. step=200 vs, step=2000

Discussion question: what headline message(s) come from each step choice? Which do you prefer?

Density plot

An alternative to a histogram for exploring frequency in continuous variable: density plot using transform_density

alt.Chart(penguins).transform_density(
    'body_mass_g',
    as_=['body_mass_g2', 'density']
).mark_area().encode(
    alt.X('body_mass_g2:Q', title = "Body Mass (g)"),
    alt.Y('density:Q', title = "Density")
)

Back to diamonds, focus on carat

alt.data_transformers.disable_max_rows()  #disable 5k max rows

alt.Chart(diamonds).mark_bar().encode(
    alt.X('carat', bin=alt.Bin(maxbins=10), title = "Carat"),
    alt.Y('count()', title = "Count")
)

First plot iteration reveals most of sample is < 2

Histogram of carat

diamonds_small = diamonds.loc[diamonds['carat'] < 2.1]

alt.Chart(diamonds_small).mark_bar().encode(
    alt.X('carat', bin=alt.BinParams(step=0.2), title = "Carat"),
    alt.Y('count()', title = "Count")
)

Second plot iteration reveals count is not entirely decreasing in carat

In-class exercise: histogram of carat

alt.Chart(diamonds_small).mark_bar().encode(
    alt.X('carat', bin=alt.BinParams(step=0.02), title = "Carat"),
    alt.Y('count()', title = "Count"))

Discussion questions

1. What is the headline of the 3rd plot iteration? Submessages?
2. What questions does it raise?

Typical continuous variables: summary

• Main tool to explore uni-dimensional continuous variables: histograms
• Varying the bin widths can reveal different patterns

Unusual continuous variables: roadmap

• case study: y dimension in diamonds
  • explore some unusual values
  • three options for handling unusual values

diamonds: identify unusual y values

First pass to examine for unusual values: summary statistics

diamonds['y'].describe()

count    53940.000000
mean         5.734526
std          1.142135
min          0.000000
25%          4.720000
50%          5.710000
75%          6.540000
max         58.900000
Name: y, dtype: float64

diamonds: examine unusual y values

diamonds.loc[(diamonds['y'] < 3) ] 

diamonds: examine unusual y values

diamonds.loc[(diamonds['y'] > 20)] 

diamonds: compare to 10 random obs

diamonds.sample(n=10)

What to do with unusual values?

1. Drop row
2. Code value to NA
3. Winsorize value

Option 1: drop rows

diamonds_clean = diamonds.loc[(diamonds['y'] >= 3) | (diamonds['y'] <= 20)] 
diamonds_clean

Option 2: recode to missing

diamonds_missing = diamonds.copy()
diamonds_missing['y'] = np.where((diamonds_missing['y'] < 3) |
     (diamonds_missing['y'] > 20), 
     np.nan, diamonds_missing['y'])
diamonds_missing[diamonds_missing['y'].isna()]

Option 3: winsorize

Winsorizing re-codes outliers to a numeric value, keeping them in the data.

To winsorize at 1 percent:

• Replace anything less than the 1st percentile with the 1st percentile
• Replace anything more than the 99th percentile with the 99th percentile

Option 3: winsorize

diamonds_winsor = diamonds.copy()
pctile01 = diamonds_winsor['y'].quantile(0.01)
pctile99 = diamonds_winsor['y'].quantile(0.99)

print(f"1st Percentile: {pctile01}")
print(f"99th Percentile: {pctile99}")

1st Percentile: 4.04
99th Percentile: 8.34

Option 3: winsorize

diamonds_winsor['y_winsor'] = np.where(diamonds_winsor['y'] < pctile01, pctile01, 
                                np.where(diamonds_winsor['y'] > pctile99, pctile99, 
                                diamonds_winsor['y']))
diamonds_winsor

When might you winsorize?

An example from Earnings Instability paper by Ganong and coauthors.

The paper is trying to quantify how much earnings change from month to month for the typical US worker.

Consider the following fake data (next slide)

Toy winsorization example

Real-world winsorization example

Source: Table A-2 from Earnings Instability paper

Pros + cons of each option

• Dropping the observation
  • Does not manipulate the data values
  • But can’t use that observation at all in your analysis
• Recoding to missing
  • Manipulates the data values
  • Allows you to use that observation – just not that variable – in your analysis
• Winsorizing
  • Manipulates the data values
  • Allows you to use that observation + that variable in your analysis

diamonds: what would you do?

• What would you do where x, y, and z are all 0?
• What would you do where y > 20?

diamonds: what would we do?

There is often not a “right” answer or you won’t know the answer without talking to a data provider.

Our best guesses:

• Rows where x, y, and z are all zero: set to NA
• Rows where y > 20: winsorize? (hard to know for sure…)

Unusual continuous values: summary

How do I decide which problem I have? Examine unusual values in context of other columns (same row) and other rows (same columns).

How do I decide whether to set to NA or winsorize? Ideally, ask your data provider what’s going on with these values.

Unusual values case study: roadmap

• Introduce mpg dataset

• Research question 1

  What is the relationship between engine size and gas mileage?”

• Research question 2

  Why do some cars have better than typical mileage?

• Ad hoc identification of outliers

• Inspect fields describing outliers

• Uncover pattern

Introducing the mpg dataset

• manufacturer — car maker (e.g., toyota, ford)
  
• model — specific model name
  
• displ — engine size (liters)
  
• hwy — gas mileage highway miles per gallon
  
• class — vehicle class (compact, suv, pickup, etc.)

Introducing the mpg dataset

Q1: What is the relationship between engine size and gas mileage?

base = alt.Chart(mpg).mark_point().encode(
    alt.X('displ:Q', title = "Engine size (displ)"),
    alt.Y('hwy:Q', title = "Gas mileage (hwy)")
)
base

Q1: What is the relationship between engine size and gas mileage?

Q2: Why do some cars have better than typical mileage?

potential_outliers = mpg.loc[(mpg["hwy"] > 40) 
    | ((mpg["hwy"] > 20) 
    & (mpg["displ"] > 5))]

Q2: Why do some cars have better than typical mileage? (plot)

Q2: In-class exercise (table)

Discussion q – which fields do you want to study further on the plot (and why?)

['manufacturer', 'model', 'displ', 'year', 'cyl', 'trans', 'drv', 'cty', 'hwy', 'fl', 'class']

Let’s focus on two fields

Fields

1. model
2. class

How did I know to use these? Context knowledge about different types of cars.

Don’t have context knowledge about your dataset? Use LLM/Google/human subject matter expert to help you identify patterns

Q2: Why do some cars have better than typical mileage?

Q: How are there big engines and good mileage? color

Discussion

• We applied labels to each outlier car using the model field
• We hypothesized that the field class would capture what these models would have in common
• This is the elegant corner case where one variable (class) explains many of the outlier patterns in terms of fuel-efficient cars
  • All the large engines on the top-right are class == “2seater”
  • Both of the small engines on the top-left are class == “subcompact” (but many subcompacts are less fuel efficient)
• Caveat: most datasets are not as clean as this example is, but we’ve chosen this example for instructional purposes

Unusual values case study: summary

Research question

Why do some cars have better than typical mileage? (What’s going on with these outliers?)

What did we do?

1. We identified outliers by hand
2. We looked at variables for those outliers
3. We went back to the plot with a variable which we thought could provide a unified explanation for the outliers.
4. It mostly did (subcompact cars and 2 seater cars are both very fuel efficient)